A study of the dynamic relation between physiological changes and spontaneous expressions

Recent progress in Affective Computing (AC) has enabled integration of physiological cues and spontaneous expressions to reveal a subject’s emotional state. Due to the lack of an effective technique for evaluating multimodal correlations, experience and intuition play a main role in present AC studies when fusing affective cues or modalities, resulting in unexpected outcomes. This study seeks to demonstrate a dynamic correlation between two such affective cues, physiological changes and spontaneous expressions, which were obtained by a combination of stereo vision based tracking and imaging photoplethysmography (iPPG), with a designed protocol involving 20 healthy subjects. The two cues obtained were sampled into a Statistical Association Space (SAS) to evaluate their dynamic correlation. It is found that the probability densities in the SAS increase as the peaks in two cues are approached. Also the complex form of the high probability density region in the SAS suggests a nonlinear correlation between two cues. Finally the cumulative distribution on the zero time-difference surface is found to be small (<0.047) demonstrating a lack of simultaneity. These results show that the two cues have a close interrelation, that is both asynchronous and nonlinear, in which a peak of one cue heralds a peak in the other

A new engineering approach to reveal correlation of physiological change and spontaneous expression from video images

Spontaneous expression is associated with physiological states, i.e., heart rate, respiration, oxygen saturation (SpO2%), and heart rate variability (HRV). There have yet not sufficient efforts to explore correlation of physiological change and spontaneous expression. This study aims to study how spontaneous expression is associated with physiological changes with an approved protocol or through the videos provided from Denver Intensity of Spontaneous Facial Action Database. Not like a posed expression, motion artefact in spontaneous expression is one of evitable challenges to be overcome in the study. To obtain a physiological signs from a region of interest (ROI), a new engineering approach is being developed with an artefact-reduction method consolidated 3D active appearance model (AAM) based track, affine transformation based alignment with opto-physiological mode based imaging photoplethysmography. Also, a statistical association spaces is being used to interpret correlation of spontaneous expressions and physiological states including their probability densities by means of Gaussian Mixture Model. The present work is revealing a new avenue of study associations of spontaneous expressions and physiological states with its prospect of applications on physiological and psychological assessment